Image processing apparatus, image processing method, and recording medium

ABSTRACT

An image processing apparatus, comprising: one or more processors; and a memory storing instructions which, when the instructions are executed by the processors, cause the image processing apparatus to function as: a detection unit configured to detect an unnecessary component generating region, which is a region of an image in which an unnecessary component is generated, based on a first viewpoint image and a second viewpoint image with different viewpoints, the image being obtained by combining the first viewpoint image and the second viewpoint image, the detection unit being configured to detect the unnecessary component generating region based on a plurality of correlation values between a first region of interest in the first viewpoint image and a plurality of second regions of interest in the second viewpoint image; and a reduction unit configured to perform processing of reducing the unnecessary component.

BACKGROUND OF THE INVENTION Field of the Invention

This disclosure relates to an image processing apparatus, an imageprocessing method, and a recording medium.

Description of the Related Art

There are cases where a part of light entering a lens forming an imagepickup optical system is reflected by an interface of the lens or amember holding the lens, to thereby reach an image pickup surface. Suchunnecessary light generates an unnecessary component, for example, ghostor flare, in an image. There are also cases where, when a diffractiveoptical element is used to correct axial chromatic aberration andlateral chromatic aberration, light from an object having a highluminance and being located outside an image pickup field angle, forexample, the sun, is diffracted by the diffractive optical element togenerate the unnecessary component in the image. Under suchcircumstances, there have been proposed methods for reducing theunnecessary light. In Japanese Patent Application Laid-Open No.2008-54206, there is disclosed a method of detecting ghost based on adifference between an image in a focused state and an image in ade-focused state. Further, in Japanese Patent No. 5284306, there isdisclosed a method of detecting ghost by comparing a plurality ofviewpoint images taken by single-lens stereoscopic imaging.

However, in the related art, there have been cases where the unnecessarycomponent cannot always be reduced satisfactorily, and where a largeprocessing load is generated.

SUMMARY OF THE INVENTION

According to an aspect of an embodiment, there is provided an imageprocessing apparatus, comprising: one or more processors; and a memorystoring instructions which, when the instructions are executed by theprocessors, cause the image processing apparatus to function as: adetection unit configured to detect an unnecessary component generatingregion, which is a region of an image in which an unnecessary componentis generated, based on a first viewpoint image and a second viewpointimage with different viewpoints, the image being obtained by combiningthe first viewpoint image and the second viewpoint image, the detectionunit being configured to detect the unnecessary component generatingregion based on a plurality of correlation values between a first regionof interest in the first viewpoint image and a plurality of secondregions of interest in the second viewpoint image; and a reduction unitconfigured to perform processing of reducing the unnecessary component.

According to another aspect of an embodiment, there is provided an imageprocessing method, comprising: detecting an unnecessary componentgenerating region, which is a region of an image in which an unnecessarycomponent is generated, based on a first viewpoint image and a secondviewpoint image with different viewpoints, the image being obtained bycombining the first viewpoint image and the second viewpoint image, theunnecessary component generating region being detected based on aplurality of correlation values between a first region of interest inthe first viewpoint image and a plurality of second regions of interestin the second viewpoint image, which are located in a vicinity of aregion corresponding to the first region of interest; and performingprocessing of reducing the unnecessary component.

According to further another aspect of an embodiment, there is provideda non-transitory computer-readable storage medium having stored thereona program for causing a computer to execute: detecting an unnecessarycomponent generating region, which is a region of an image in which anunnecessary component is generated, based on a first viewpoint image anda second viewpoint image with different viewpoints, the image beingobtained by combining the first viewpoint image and the second viewpointimage, the unnecessary component generating region being detected basedon a plurality of correlation values between a first region of interestin the first viewpoint image and a plurality of second regions ofinterest in the second viewpoint image, which are located in a vicinityof a region corresponding to the first region of interest; andperforming processing of reducing the unnecessary component.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram for illustrating an image processing apparatusaccording to a first embodiment of this disclosure.

FIG. 2 is a diagram for illustrating a relationship between pupilregions of an image pickup optical system and a light receiving portionof an image pickup element.

FIG. 3 is a diagram for illustrating a relationship between the pupilregions of the image pickup optical system and the light receivingportion of the image pickup element.

FIG. 4A and FIG. 4B are diagrams for illustrating the structure of theimage pickup optical system and unnecessary light.

FIG. 5 is a diagram for conceptually illustrating an aperture of adiaphragm.

FIG. 6 is a diagram for illustrating an example of an image taken byphotography.

FIG. 7A and FIG. 7B are diagrams for illustrating examples of viewpointimages.

FIG. 8 is a flow chart for illustrating an operation of the imageprocessing apparatus according to the first embodiment.

FIG. 9 is a diagram for illustrating an example of a difference valuemap.

FIG. 10 is a flow chart for illustrating an operation of the imageprocessing apparatus according to the first embodiment.

FIG. 11A and FIG. 11B are diagrams for each illustrating a pixel ofinterest and a region of interest.

FIG. 12A, FIG. 12B, FIG. 12C, and FIG. 12D are diagrams for illustratingexamples of a sum of absolute differences.

FIG. 13 is a diagram for illustrating a reduced intensity value map.

FIG. 14 is a diagram for illustrating an image in which an unnecessarycomponent is reduced.

FIG. 15 is a diagram for illustrating an example of an image taken byphotography.

FIG. 16A and FIG. 16B are diagrams for illustrating examples ofviewpoint images.

FIG. 17 is a flow chart for illustrating an operation of an imageprocessing apparatus according to a second embodiment of thisdisclosure.

FIG. 18 is a diagram for illustrating an unnecessary component amountmap.

FIG. 19 is a flow chart for illustrating an operation of the imageprocessing apparatus according to the second embodiment.

FIG. 20 is a diagram for illustrating an image in which an unnecessarycomponent is reduced.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail in accordance with the accompanying drawings.

First Embodiment

An image processing apparatus and an image processing method accordingto a first embodiment of this disclosure are described with reference toFIG. 1 to FIG. 14. An image processing apparatus 100 according to thefirst embodiment is an image pickup apparatus capable of taking aplurality of images having parallax therebetween, that is, a pluralityof viewpoint images, for example. Here, there is described, as anexample, a case where the image processing apparatus 100 is an imagepickup apparatus, that is, a case where the image processing apparatus100 includes an image pickup element 102, but the image processingapparatus 100 may include no image pickup element 102. In other words,the image processing apparatus 100 may perform image processing using aplurality of viewpoint images taken by an image pickup apparatus that isseparate from the image processing apparatus 100.

FIG. 2 is a diagram for illustrating a relationship between pupilregions of an image pickup optical system and a light receiving portionof the image pickup element. An exit pupil EXP of an image pickupoptical system 101 (see FIG. 1) includes a plurality of mutuallydifferent pupil regions, that is, a first pupil region P1 and a secondpupil region P2. In a light receiving portion (light receiving surface)114 of the image pickup element 102, unit pixels (pixel pairs) 115, eachof which includes a plurality of pixels (photoelectric converters,divided pixels, subpixels), that is, a first pixel G1 and a second pixelG2, are arranged in matrix. In other words, in the light receivingportion 114 of the image pickup element 102, there is provided a pixelarray in which the plurality of unit pixels 115 are arranged in matrix.A color filter CF is arranged above each of the unit pixels 115, and amicrolens ML is arranged above the color filter CF. The first pixel G1and the second pixel G2 included in one unit pixel 115 form a pair. Inother words, the first pixel G1 and the second pixel G2 included in theone unit pixel 115 share one microlens ML corresponding to the unitpixel 115. The first pixel G1 and the second pixel G2 forming the pairhave a conjugate relationship with the exit pupil EXP via the microlensML. In this manner, light fluxes that pass through the mutuallydifferent pupil regions P1 and P2 of the exit pupil EXP of the imagepickup optical system 101 are guided to the mutually different pixels G1and G2 arranged in the image pickup element 102, respectively, to bephotoelectrically converted in those pixels G1 and G2, respectively.

FIG. 3 is a diagram for illustrating a relationship between the pupilregions of the image pickup optical system and the light receivingportion of the image pickup element. In FIG. 3, there is illustrated acase where it is assumed that a thin lens is provided at the position ofthe exit pupil EXP. The first pixel G1 receives the light flux that haspassed through the first pupil region P1 of the exit pupil EXP. Thesecond pixel G2 receives the light flux that has passed through thesecond pupil region P2 of the exit pupil EXP. There is no need for anobject to be always present at an object point OSP. The light flux thatpasses through the pupil region P1 of the light fluxes from the objectpoint OSP enters the first pixel G1. The light flux that passes throughthe pupil region P2 of the light fluxes from the object point OSP entersthe second pixel G2. Receiving of the light fluxes that have passedthrough the mutually different pupil regions P1 and P2 of the exit pupilEXP by the mutually different pixels G1 and G2 included in the unitpixel 115 is called “pupil division”. The light fluxes that pass throughthe mutually different pupil regions P1 and P2 enter those pixels G1 andG2, respectively, and hence parallax is generated between an imagegenerated using a signal from the first pixels G1 and an image generatedusing a signal from the second pixels G2. In this specification, theimage generated using the signal from a plurality of the first pixels G1(first pixel group) is referred to as a first viewpoint image.Similarly, the image generated using the signal from a plurality of thesecond pixels G2 (second pixel group) is referred to as a secondviewpoint image.

There are cases where the above-mentioned conjugate relationship maybecome incomplete due to a positional deviation of the exit pupil EXPand other such causes. There are also cases where the first pupil regionP1 and the second pupil region P2 included in the exit pupil EXPpartially overlap with each other. Also in such cases, parallax isgenerated between the image generated by the signal from the pixels G1and the image generated by the signal from the pixels G2. Therefore, theabove-mentioned conjugate relationship does not always need to becomplete. Moreover, the first pupil region P1 and the second pupilregion P2 included in the exit pupil EXP may partially overlap with eachother.

FIG. 1 is a block diagram for illustrating the image processingapparatus according to the first embodiment. The image pickup opticalsystem 101 forms an optical image, that is, an object image from anobject (not shown) on the light receiving portion 114 of the imagepickup element 102. As the image pickup element 102, a charge-coupleddevice (CCD) image sensor or a complementary metal oxide semiconductor(CMOS) image sensor is used, for example. The image pickup element 102receives the light fluxes that pass through the mutually different pupilregions P1 and P2 of the exit pupil EXP with the pixels G1 and G2,respectively. The image pickup element 102 photoelectrically convertsthe object image and outputs an analog image signal. An A/D converter103 converts the analog image signal output from the image pickupelement 102 into a digital image signal, and outputs the digital imagesignal to an image processing unit 104.

The image processing unit 104 performs image processing on the digitalimage signal from the A/D converter 103. As the image processing unit104, a digital signal processor (DSP) is used, for example. As describedlater, the image processing unit 104 acquires a plurality of viewpointimages with different viewpoints, specifically, a first viewpoint image117 a (see FIG. 7A) and a second viewpoint image 117 b (see FIG. 7B). Asdescribed later, the first viewpoint image 117 a is an imagecorresponding to a first image signal obtained by the plurality of firstpixels G1. Similarly, as described later, the second viewpoint image 117b is an image corresponding to a second image signal obtained by theplurality of second pixels G2. As described later, the image processingunit 104 detects an unnecessary component generating region GST, whichis a region of an image 117 (see FIG. 6) in which an unnecessarycomponent is generated, based on the plurality of viewpoint images 117 aand 117 b, the image 117 corresponding to a combined signal of the firstimage signal and the second image signal. As described later, the imageprocessing unit 104 detects the unnecessary component generating regionGST based on a plurality of correlation values between a first region ofinterest ra in the first viewpoint image 117 a and a plurality of secondregions of interest rb in the second viewpoint image 117 b (see FIG. 11Aand FIG. 11B). As described later, the plurality of second regions ofinterest rb are set by sequentially shifting the second region ofinterest rb in a predetermined range including a region corresponding tothe first region of interest ra. As such correlation value, a sum ofabsolute differences (SAD) is used, for example. More specifically, asum of absolute differences SAD between a portion located in the firstregion of interest ra of the first viewpoint image 117 a and a portionlocated in the second region of interest rb of the second viewpointimage 117 b is used as such correlation value, for example. As describedlater, the image processing unit 104 detects the unnecessary componentgenerating region GST based on comparison between the plurality ofcorrelation values and a threshold value THR. As described later, thethreshold value THR is set based on a contrast in the first region ofinterest ra. Alternatively, as described later, the threshold value THRis set based on an average value of pixel values in the first region ofinterest ra. Still alternatively, as described later, the thresholdvalue THR is set based on an ISO sensitivity at the time of photography.In this manner, the image processing unit 104 serves as a detection unitconfigured to detect the unnecessary component generating region GST.The image processing unit 104 also serves as a reduction unit configuredto perform processing of reducing the unnecessary component. Asdescribed later, the image processing unit 104 selectively performs theprocessing of reducing the unnecessary component on the unnecessarycomponent generating region GST. As described later, the imageprocessing unit 104 reduces the unnecessary component so that a degreeof reduction of the unnecessary component is gradually changed at aboundary between the unnecessary component generating region GST and anunnecessary component non-generating region, which is a region in whichthe unnecessary component is not generated. The image processing unit104 also performs general image processing, for example, gammacorrection and color balancing, on the image in which the unnecessarycomponent has been reduced, to thereby generate an image file in apredetermined file format. The image processing unit 104 detects theunnecessary component generating region GST based on the first viewpointimage 117 a and the second viewpoint image 117 b, on which correction ofan image shift depending on the parallax has not been performed.Examples of the predetermined file format include the Joint PhotographicExperts Group (JPEG) format.

The image file (image data) generated by the image processing unit 104is stored in a recording medium 109. Examples of the recording medium109 include a semiconductor memory and an optical disc. The recordingmedium 109 may or may not be detachable from the image processingapparatus 100. Further, an image output from the image processing unit104 may be displayed by a display 105. A storage unit (memory) 108stores an image processing program, which is required for the imageprocessing performed by the image processing unit 104, and various kindsof information. The storage unit 108 includes a random access memory(RAM) and a read only memory (ROM), for example.

A system controller (controller) 110 is configured to control the imagepickup element 102 and control the image processing unit 104. As thesystem controller 110, a central processing unit (CPU) is used, forexample. The system controller 110 is also configured to control adiaphragm 101 a and a focus lens 101 b included in the image pickupoptical system 101 via an image optical system controller 106. The imageoptical system controller 106 controls an aperture diameter of thediaphragm 101 a based on an aperture value (f-number) specified by thesystem controller 110. The image optical system controller 106 alsoadjusts the focus lens 101 b to be focused on an object based on aninstruction from the system controller 110 (autofocus). A user maymanually operate the focus lens 101 b to be focused on the object(manual focus). A state detection unit 107 is configured to acquirecurrent photographic conditions and other such information in responseto an instruction from the system controller 110. The image pickupoptical system 101 may or may not be detachable from a main body (body)of the image processing apparatus 100.

FIG. 4A and FIG. 4B are diagrams for illustrating the structure of theimage pickup optical system 101 and unnecessary light. In FIG. 4A, thereis illustrated a state in which unnecessary light does not enter theimage pickup element 102, and in FIG. 4B, there is illustrated a statein which unnecessary light enters the image pickup element 102. Asillustrated in FIG. 4B, strong light from the sun SUN, which is anobject having a high luminance, enters lenses forming the image pickupoptical system 101, and light reflected by interfaces of the lenses,that is, unnecessary light 121 reaches the light receiving portion 114of the image pickup element 102. The unnecessary light 121 generates theunnecessary component, for example, ghost or flare, in an image. Here,as illustrated in FIG. 4B, there is described, as an example, a casewhere the unnecessary light 121 passes through a first portion 116 a ofthe aperture 116 of the diaphragm 101 a, while the unnecessary light 121does not pass through a second portion 116 b of the aperture 116 of thediaphragm 101 a.

FIG. 5 is a diagram for conceptually illustrating the aperture 116 ofthe diaphragm 101 a. The first portion 116 a of the aperture 116 of thediaphragm 101 a corresponds to the first pupil region P1 of the exitpupil EXP. Meanwhile, the second portion 116 b of the aperture 116 ofthe diaphragm 101 a corresponds to the second pupil region P2 of theexit pupil EXP. The aperture 116 of the diaphragm 101 a and the exitpupil EXP of the image pickup optical system 101 are different in astrict sense. As described above, the unnecessary light 121 passesthrough the first portion 116 a of the aperture 116 of the diaphragm 101a. As described above, the first portion 116 a corresponds to the firstpupil region P1. Moreover, as described above, the first pixel G1receives the light flux that passes through the first pupil region P1.Therefore, the unnecessary light 121 reaches the first pixel G1.Meanwhile, as described above, the unnecessary light 121 does not passthrough the second portion 116 b of the aperture 116 of the diaphragm101 a. As described above, the second portion 116 b corresponds to thesecond pupil region P2. Moreover, as described above, the second pixelG2 receives the light flux that passes through the second pupil regionP2. Therefore, the unnecessary light 121 does not reach the second pixelG2. In this manner, in the example illustrated in FIG. 4A and FIG. 4B,the unnecessary light 121 reaches the first pixel G1, but does not reachthe second pixel G2.

FIG. 6 is a diagram for illustrating an example of an image taken byphotography. The image illustrated in FIG. 6 is generated from acombined signal obtained by combining the image signal acquired by thefirst pixels G1 and the image signal acquired by the second pixels G2,for example. As illustrated in FIG. 6, in an image 117, objects 118 and119, that is, flowers are arranged in the front center and in the backleft, respectively. Moreover, in the image 117, the unnecessarycomponent generating region GST, that is, a region in which ghost or thelike is generated is present. In the unnecessary component generatingregion GST, the unnecessary component overlaps with the object, andhence a pixel value (luminance) in the unnecessary component generatingregion GST is higher than an original pixel value of the object.

FIG. 7A and FIG. 7B are diagrams for illustrating examples of theviewpoint images. In FIG. 7A, the first viewpoint image 117 a isillustrated, and in FIG. 7B, the second viewpoint image 117 b isillustrated. The first viewpoint image 117 a illustrated in FIG. 7A isan image obtained by photoelectrically converting the light fluxes thatpass through the first pupil region P1 by the first pixel group formedof the plurality of first pixels G1. As described above, the unnecessarylight 121 reaches the first pixel G1. Therefore, the first viewpointimage 117 a illustrated in FIG. 7A includes the unnecessary componentgenerating region GST, that is, the region in which ghost or the like isgenerated. Meanwhile, the second viewpoint image 117 b illustrated inFIG. 7B is an image obtained by photoelectrically converting the lightfluxes that pass through the second pupil region P2 by the second pixelgroup formed of the plurality of second pixels G2. As described above,the unnecessary light 121 does not reach the second pixel G2. Therefore,the second viewpoint image 117 b illustrated in FIG. 7B does not includethe unnecessary component generating region GST. The first pixel G1 andthe second pixel G2 are arranged to be adjacent to each other in ahorizontal direction, and hence parallax in the left-and-right directionis generated between the first viewpoint image 117 a illustrated in FIG.7A and the second viewpoint image 117 b illustrated in FIG. 7B.Moreover, here, there is described, as an example, a case where thenumber of unnecessary component generating regions GST is one, but theunnecessary component generating regions GST may be present at aplurality of spots in an image. Moreover, the unnecessary componentgenerating regions GST may overlap with each other in an image. At aspot at which unnecessary components overlap with each other, a pixelvalue (luminance) becomes higher than at a spot at which unnecessarycomponents do not overlap with each other. There are a differencecorresponding to the unnecessary component and a differencecorresponding to an angular difference (amount of parallax) in aline-of-sight direction, that is, a difference corresponding to theparallax between the first viewpoint image 117 a illustrated in FIG. 7Aand the second viewpoint image 117 b illustrated in FIG. 7B.

FIG. 8 is a flow chart for illustrating an operation of the imageprocessing apparatus according to the first embodiment.

In Step S201, the image processing unit 104 acquires the plurality ofviewpoint images having parallax therebetween (having differentviewpoints), that is, the first viewpoint image 117 a and the secondviewpoint image 117 b. The image processing unit 104 performs generaldevelopment processing and various kinds of correction processing oneach of the first viewpoint image 117 a and the second viewpoint image117 b. After Step S201, Step S202 and Step S203 are performed inparallel.

In Step S202, the image processing unit 104 calculates a differencevalue between the first viewpoint image 117 a and the second viewpointimage 117 b. More specifically, the image processing unit 104 calculatesan absolute value of a difference between the first viewpoint image 117a and the second viewpoint image 117 b for each set of coordinates. Whena pixel value at each set of coordinates of the first viewpoint image117 a is represented by A(x,y), a pixel value at each set of coordinatesof the second viewpoint image 117 b is represented by B(x,y), and adifference value at each set of coordinates is represented by DIFF(x,y),a relationship expressed as the following expression (1) is established:

DIFF(x,y)=|A(x,y)−B(x,y)|  (1).

FIG. 9 is a diagram for illustrating an example of a difference valuemap. A difference value map 117 c illustrated in FIG. 9 is obtained bytwo-dimensionally mapping the difference value DIFF(x,y) at each set ofcoordinates between the first viewpoint image 117 a illustrated in FIG.7A and the second viewpoint image 117 b illustrated in FIG. 7B. In thedifference value map 117 c, the difference corresponding to theunnecessary component and the difference corresponding to the parallaxappear. The object 118 in the front of the two objects 118 and 119included in the image 117 illustrated in FIG. 6 is in focus, and hencethe difference corresponding to the parallax does not appearconspicuously in a portion corresponding to the object 118. Meanwhile,the object 119 located in the back left of the two objects 118 and 119included in the image 117 illustrated in FIG. 6 is out of focus, andhence the difference corresponding to the parallax appears conspicuouslyin a portion corresponding to the object 119.

In Step S203, the image processing unit 104 generates a reducedintensity value, which indicates a reduced intensity of the unnecessarycomponent, based on the first viewpoint image 117 a and the secondviewpoint image 117 b. FIG. 10 is a flow chart for illustrating anoperation of the image processing apparatus according to the firstembodiment. In FIG. 10, there is illustrated processing of generatingthe reduced intensity value, which is performed by the image processingapparatus according to the first embodiment. The processing ofgenerating the reduced intensity value is sequentially performed foreach of a plurality of pixels included in an image. In Step S301, theimage processing unit 104 determines whether or not the processing ofgenerating the reduced intensity value has been completed for allpixels. When the processing of generating the reduced intensity valuehas not been completed for all pixels (NO in Step S301), the processingproceeds to Step S302. Meanwhile, when the processing of generating thereduced intensity value has been completed for all pixels (YES in StepS301), the processing proceeds to Step S306.

FIG. 11A and FIG. 11B are diagrams for each illustrating a pixel ofinterest and a region of interest. In FIG. 11A, a part of the firstviewpoint image 117 a is illustrated while being enlarged, and onesquare in FIG. 11A corresponds to one pixel. A first pixel of interestpa is a pixel that is a current target of the processing of generatingthe reduced intensity value of a plurality of pixels included in thefirst viewpoint image 117 a. The first region of interest ra is set inthe first viewpoint image 117 a with the first pixel of interest pabeing the center. Here, the first region of interest ra has a size of 9pixels by 9 pixels. However, the size of the first region of interest rais not limited thereto, and may be set as appropriate. In FIG. 11B, apart of the second viewpoint image 117 b is illustrated while beingenlarged, and one square in FIG. 11B corresponds to one pixel. A set ofcoordinates of a second pixel of interest pb illustrated in FIG. 11Bcorresponds to a set of coordinates of the first pixel of interest paillustrated in FIG. 11A. Moreover, a position of the second region ofinterest rb illustrated in FIG. 11B corresponds to a position of thefirst region of interest ra illustrated in FIG. 11A. The size of thesecond region of interest rb is set to be equivalent to the size of thefirst region of interest ra.

In Step S302, the image processing unit 104 acquires the sum of absolutedifferences SAD between the portion in the first region of interest raof the first viewpoint image 117 a and the portion in the second regionof interest rb of the second viewpoint image 117 b while shifting thesecond region of interest rb. More specifically, the sum of absolutedifferences SAD between a pixel value of each of a plurality of pixelslocated in the first region of interest ra and a pixel value of each ofa plurality of pixels located in the second region of interest rb isacquired. A direction of shifting the second region of interest rb is adirection in which the parallax is generated between the first viewpointimage 117 a and the second viewpoint image 117 b, that is, theleft-and-right direction in this example. The arrows illustrated in FIG.11B indicate directions of shifting the second region of interest rb.

FIG. 12A to FIG. 12D are diagrams for illustrating examples of resultsof calculating the sum of absolute differences SAD. FIG. 12A is adiagram for illustrating a first set of coordinates pos1 and a secondset of coordinates pos2 in an image. In FIG. 12B, there is illustratedan example of a result of calculating the sum of absolute differencesSAD when the first pixel of interest pa is located at the first set ofcoordinates pos1. In FIG. 12C, there is illustrated an example of aresult of calculating the sum of absolute differences SAD when the firstpixel of interest pa is located at the second set of coordinates pos2.The horizontal axis in each of FIG. 12B and FIG. 12C indicates a shiftamount of the second region of interest rb. When the set of coordinatesof the first pixel of interest pa and the set of coordinates of thesecond pixel of interest pb is the same, the shift amount indicated bythe horizontal axis in FIG. 12B and FIG. 12C is 0.

As illustrated in FIG. 12A, the first set of coordinates pos1 is locatedin the unnecessary component generating region GST. Therefore, when thefirst pixel of interest pa is located at the first set of coordinatespos1, the first pixel of interest pa is located in the unnecessarycomponent generating region GST. Here, there is described, as anexample, a case where the object is flat in the vicinity of the firstset of coordinates pos1. A difference generated between the portion inthe first region of interest ra of the first viewpoint image 117 a andthe portion in the second region of interest rb of the second viewpointimage 117 b is mainly caused by the unnecessary component, and not bythe parallax. Therefore, in a case where the first pixel of interest pais located at the first set of coordinates pos1, as illustrated in FIG.12B, even when the second region of interest rb is shifted within thepredetermined range, the sum of absolute differences SAD is hardlychanged. Moreover, even when the second region of interest rb is shiftedwithin the predetermined range, the sum of absolute differences SAD doesnot fall below the threshold value THR as illustrated in FIG. 12B.Meanwhile, in a case where the first pixel of interest pa is located atthe second set of coordinates pos2, a difference caused by the parallaxis generated conspicuously between the first viewpoint image 117 a andthe second viewpoint image 117 b. Therefore, when the second region ofinterest rb is shifted, the sum of absolute differences SAD variessignificantly, and falls below the threshold value THR. In this manner,in a case where the first pixel of interest pa is located in theunnecessary component generating region GST, that is, in a case wherethe first region of interest ra is located in the unnecessary componentgenerating region GST, the following result is generated. Specifically,there is a low correlation between the pixel values in the first regionof interest ra and the pixel values in the second region of interest rb,and hence the sum of absolute differences SAD becomes higher than thethreshold value THR. In addition, even when the second region ofinterest rb is shifted within the predetermined range, the sum ofabsolute differences SAD does not fall below the threshold value THR.Therefore, even when the second region of interest rb is shifted withinthe predetermined range, and when the sum of absolute differences SADdoes not fall below the threshold value THR, it can be determined thatthe first pixel of interest pa is located within the unnecessarycomponent generating region GST.

In Step S303, the image processing unit 104 determines whether or not aminimum value of the sums of absolute differences SAD obtained when thesecond region of interest rb is shifted within the predetermined rangeis lower than the threshold value THR. When the minimum value of thesums of absolute differences SAD obtained when the second region ofinterest rb is shifted within the predetermined range is lower than thethreshold value THR (YES in Step S303), it is considered that the firstpixel of interest pa is not located within the unnecessary componentgenerating region GST. In such case, it is considered that there is noneed to perform the processing of reducing the unnecessary component,and hence a reduced intensity for the first pixel of interest pa is setto 0, for example. In this case, a reduced intensity value DEC(x,y) atthe set of coordinates of the first pixel of interest pa is expressed asfollows:

DEC(x,y)=0  (2).

In Step S304, the image processing unit 104 records 0 as a reducedintensity value for the first pixel of interest pa in a reducedintensity value map. The reduced intensity value map is obtained bytwo-dimensionally mapping the reduced intensity value DEC(x,y) for eachfirst pixel of interest pa. When the minimum value of the sums ofabsolute differences SAD obtained when shifting the second region ofinterest rb within the predetermined range is the threshold value THR ormore (NO in Step S303), it is considered that the first pixel ofinterest pa is located in the unnecessary component generating regionGST. In this case, it is considered that there is a need to perform theprocessing of reducing the unnecessary component, and hence a reducedintensity for the first pixel of interest pa is set to 255, for example.In this case, the reduced intensity value DEC(x,y) at the set ofcoordinates of the first pixel of interest pa is expressed as follows:

DEC(x,y)=255  (3).

In Step S305, the image processing unit 104 records 255 as the reducedintensity for the first pixel of interest pa in the reduced intensityvalue map.

A portion in which the reduced intensity value is 0 in the reducedintensity value map corresponds to the region in which the unnecessarycomponent is not generated. A portion in which the reduced intensity is255 in the reduced intensity value map corresponds to the unnecessarycomponent generating region GST. Such reduced intensity value map isstored in the storage unit 108, for example. Here, there has beendescribed, as an example, the case where the reduced intensity value isset to 0 or 255. However, the reduced intensity value is not limitedthereto, and may be set as appropriate.

The threshold value THR is set based on the following expression (4),for example:

THR=K×CNT×AVR+OFS  (4).

In the expression (4), K represents an adjustment factor. CNT representsa contrast in the first region of interest ra, for example, a differencebetween a maximum value and a minimum value of the pixel values of thepixels located in the first region of interest ra. AVR represents anaverage value of the pixel values of the pixels located in the firstregion of interest ra. OFS represents an offset value, and is setdepending on the ISO sensitivity at the time when an image is taken, forexample.

When there is a steep edge or the like in the regions of interest ra andrb, and hence the pixel values vary significantly in the regions ofinterest ra and rb, the minimum value of the sums of absolutedifferences SAD tends to be large even though the unnecessary componentis not generated in the regions of interest ra and rb. Moreover, as thecontrast of the image in the regions of interest ra and rb becomeshigher, the difference between the pixel values of the region ofinterest ra and the pixel values of the region of interest rb becomeslarger, and hence the sum of absolute differences SAD tends to becomelarger. Moreover, when a difference is generated between the signalobtained by the first pixel G1 and the signal obtained by the secondpixel G2 due to an angle of light entering the microlens ML, as anintensity of the light becomes higher, the difference becomes larger,and hence the sum of absolute differences SAD becomes larger. Therefore,the threshold value THR may be set based on the contrast CNT and theaverage value AVR to contribute to prevention of false detection of theunnecessary component generating region GST. When the ISO sensitivity atthe time when the image is taken is high, the sum of absolutedifferences SAD between the first region of interest ra and the secondregion of interest rb is large due to effects of noise. Therefore, theoffset OFS is set as appropriate based on the ISO sensitivity at thetime when the image is taken. The adjustment factor K is set based on aresult of simulation or an experiment, for example. As described above,in the first embodiment, the minimum value of the sums of absolutedifferences SAD may be compared with the threshold value THR to detectthe unnecessary component generating region GST.

Here, there has been described, as an example, the case where the regionin which the difference caused by the unnecessary component is generatedbetween the plurality of viewpoint images 117 a and 117 b and the regionin which the difference caused by the parallax is generated between theplurality of viewpoint images 117 a and 117 b are different. However,both of the difference caused by the unnecessary component and thedifference caused by the parallax may be generated at the same spot.Even when the difference caused by the unnecessary component and thedifference caused by the parallax are generated at the same spot, theunnecessary component generating region GST may be detected in the samemanner as described above. In the case where both of the differencecaused by the unnecessary component and the difference caused by theparallax are generated at the same spot, a sum of absolute differencesSAD obtained by adding the sum of absolute differences SAD illustratedin FIG. 12B and the sum of absolute differences SAD illustrated in FIG.12C is obtained. In FIG. 12D, there is illustrated an example of aresult of calculating the sum of absolute differences SAD when both ofthe difference caused by the unnecessary component and the differencecaused by the parallax are generated at the same spot. As illustrated inFIG. 12D, the sum of absolute differences SAD varies depending on theshift amount of the second region of interest rb, but the minimum valueof the sums of absolute differences SAD does not fall below thethreshold value THR. Therefore, the minimum value of the sums ofabsolute differences SAD may be compared with the threshold value THR todetect the unnecessary component generating region GST. In other words,as described above, even when the second region of interest rb isshifted within the predetermined range, and when the sum of absolutedifferences SAD does not fall below the threshold value THR, it may bedetermined that the first pixel of interest pa is located within theunnecessary component generating region GST.

FIG. 13 is a diagram for illustrating a reduced intensity value map 122.The unnecessary component generating region GST, that is, a portion inwhich the reduced intensity value DEC(x,y) is 255 is shown in white inFIG. 13. The region in which the unnecessary component is not generated,that is, a portion in which the reduced intensity value DEC(x,y) is 0 isshown in black in FIG. 13.

In Step S306, the image processing unit 104 smooths the reducedintensity value DEC(x,y). The image processing unit 104 applies ageneral smoothing filter or the like to the reduced intensity value map122 so that the reduced intensity value is gradually changed at aboundary between the unnecessary component generating region GST and theregion in which the unnecessary component is not generated. As a result,a smoothed reduced intensity value DEC′(x,y) is obtained.

In Step S204, the image processing unit 104 performs the processing ofreducing the unnecessary component on the image 117 taken byphotography. An image G(x,y) taken by photography is an image generatedfrom the combined signal of the first image signal acquired by the firstpixels G1 and the second image signal acquired by the second pixels G2.When the processing of reducing the unnecessary component is performed,the reduced intensity value obtained in Step S306, that is, the smoothedreduced intensity value DEC′(x,y) is used. The processing of reducingthe unnecessary component is expressed by the following expression (5),for example. In the expression (5), G(x,y) represents an image taken byphotography, and G′(x,y) represents an image on which the processing ofreducing the unnecessary component has been performed.

G′(x,y)=G(x,y)−DIFF(x,y)×DEC′(x,y)/255  (5)

The processing of reducing the unnecessary component is selectivelyperformed on the unnecessary component generating region GST of theimage taken by photography. The processing of reducing the unnecessarycomponent is performed using the smoothed reduced intensity valueDEC′(x,y), and hence, more precisely, this processing is performed onthe unnecessary component generating region GST and the vicinity of theunnecessary component generating region GST.

FIG. 14 is a diagram for illustrating an image in which the unnecessarycomponent is reduced. As illustrated in FIG. 14, in an image 117 d inwhich the unnecessary component has been reduced, the unnecessarycomponent, that is, ghost or the like is sufficiently reduced. In thefirst embodiment, the unnecessary component is reduced using the reducedintensity value that is gradually changed at the boundary between theunnecessary component generating region GST and the region in which theunnecessary component is not generated, that is, the smoothed reducedintensity value. Therefore, according to the first embodiment,occurrence of unnaturalness in the image can be prevented at theboundary between the region in which the processing of reducing theunnecessary component has been performed and the region in which theprocessing of reducing the unnecessary component has not been performed.

As described above, in the first embodiment, the unnecessary componentgenerating region GST is detected based on the plurality of correlationvalues between the first region of interest ra in the first viewpointimage 117 a and the plurality of second regions of interest rb in thesecond viewpoint image 117 b. According to the first embodiment, theunnecessary component, for example, ghost or flare, may be reducedsatisfactorily without generating a large processing load. In addition,according to the first embodiment, the unnecessary component is reducedso that the degree of reduction of the unnecessary component isgradually changed at the boundary between the unnecessary componentgenerating region GST and the unnecessary component non-generatingregion, which is the region in which the unnecessary component is notgenerated. Therefore, according to the first embodiment, the occurrenceof unnaturalness can be prevented in the image in which the unnecessarycomponent has been removed.

Second Embodiment

An image processing apparatus and an image processing method accordingto a second embodiment of this disclosure are described with referenceto FIG. 15 to FIG. 20. The same constituent elements as those of theimage processing apparatus and the image processing method according tothe first embodiment, which are illustrated in FIG. 1 to FIG. 14, aredenoted by the same reference symbols, and a description thereof isomitted or simplified.

FIG. 15 is a diagram for illustrating an example of an image taken byphotography. The image illustrated in FIG. 15 is generated by thecombined signal obtained by combining the image signal acquired by thefirst pixels G1 and the image signal acquired by the second pixels G2,for example. As illustrated in FIG. 15, in an image 120, objects 124 and125, that is, flowers are arranged in the front center and in the backright, respectively. Moreover, in the image 120, there is an unnecessarycomponent generating region GST. The unnecessary component generatingregion GST overlaps with the object 125. The object 124 in the front ofthe two objects 124 and 125 included in the image 120 illustrated inFIG. 15 is in focus, and hence the difference corresponding to theparallax does not appear conspicuously between the plurality ofviewpoint images in a portion corresponding to the object 124.Meanwhile, the object 125 located in the back right of the two objects124 and 125 included in the image 120 illustrated in FIG. 15 is out offocus, and hence the difference corresponding to the parallax appearsconspicuously between the plurality of viewpoint images in a portioncorresponding to the object 125. In FIG. 15, the unnecessary componentgenerating region GST is located at a spot at which the differencecorresponding to the parallax appears conspicuously between theviewpoint images. In such case, when the image processing is simplyperformed by the image processing method according to the firstembodiment described above, the image may be degraded at the spot. Theimage processing apparatus according to the second embodiment is capableof suppressing such degradation of the image.

FIG. 16A and FIG. 16B are diagrams for illustrating examples ofviewpoint images. In FIG. 16A, a first viewpoint image 120 a isillustrated, and in FIG. 16B, a second viewpoint image 120 b isillustrated. The first viewpoint image 120 a illustrated in FIG. 16A isa viewpoint image obtained by photoelectrically converting the lightfluxes that pass through the first pupil region P1 by the first pixelgroup formed of the plurality of first pixels G1. As described above,the unnecessary light 121 reaches the first pixel G1. Therefore, thefirst viewpoint image 120 a illustrated in FIG. 16A includes theunnecessary component generating region GST. Meanwhile, the secondviewpoint image 120 b illustrated in FIG. 16B is an image obtained byphotoelectrically converting the light fluxes that pass through thesecond pupil region P2 by the second pixel group formed of the pluralityof second pixels G2. As described above, the unnecessary light 121 doesnot reach the second pixel G2. Therefore, the second viewpoint image 120b illustrated in FIG. 16B does not include the unnecessary componentgenerating region GST. The first pixel G1 and the second pixel G2 arearranged to be adjacent to each other in a horizontal direction, andhence parallax in the left-and-right direction is generated between thefirst viewpoint image 120 a illustrated in FIG. 16A and the secondviewpoint image 120 b illustrated in FIG. 16B. Moreover, here, there isdescribed, as an example, a case where the number of unnecessarycomponent generating regions GST is one, but the unnecessary componentgenerating regions GST may be present at a plurality of spots in animage. Moreover, the unnecessary component generating regions GST mayoverlap with each other in an image. There are a differencecorresponding to the unnecessary component and a differencecorresponding to the parallax between the first viewpoint image 120 aand the second viewpoint image 120 b.

FIG. 17 is a flow chart for illustrating an operation of the imageprocessing apparatus according to the second embodiment.

In Step S401, the image processing unit 104 acquires the plurality ofviewpoint images having parallax therebetween, that is, the firstviewpoint image 120 a and the second viewpoint image 120 b. The imageprocessing unit 104 performs general development processing and variouskinds of correction processing on each of the first viewpoint image 120a and the second viewpoint image 120 b.

In Step S402, the image processing unit 104 calculates an unnecessarycomponent amount and generates a reduced intensity value. FIG. 19 is aflow chart for illustrating an operation of the image processingapparatus according to the second embodiment. In FIG. 19, there isillustrated processing of performing the calculation of the unnecessarycomponent amount and the generation of the reduced intensity value. StepS501 to Step S505 illustrated in FIG. 19 are the same as Step S301 toStep S305 described above with reference to FIG. 10, and hence adescription thereof is omitted.

In Step S506, the image processing unit 104 generates an unnecessarycomponent amount map. The unnecessary component amount map is obtainedby two-dimensionally mapping an unnecessary component amount GHOST(x,y)for each first pixel of interest pa. A pixel value at each set ofcoordinates of the first viewpoint image 120 a is represented by A(x,y),a pixel value of each pixel of the second viewpoint image 120 b isrepresented by B(x,y), and the shift amount of the second region ofinterest rb at the time when the sum of absolute differences SAD isminimized, that is, the correlation value is maximized is represented byd. The unnecessary component amount GHOST(x,y) at each pixel isexpressed by the following expressions.

Specifically, when A(x,y)≧B(x+d,y+d),

GHOST(x,y)=A(x,y)−B(x+d,y+d)  (6), and

when A(x,y)<B(x+d,y+d),

GHOST(x+d,y+d)=B(x+d,y+d)−A(x,y)  (7).

FIG. 18 is a diagram for illustrating an unnecessary component amountmap 123. The region in which the unnecessary component is not generated,that is, a portion in which the unnecessary component amount GHOST(x,y)is 0 is shown in black in FIG. 18. In the unnecessary componentgenerating region GST, the unnecessary component amount GHOST(x,y) ateach set of coordinates takes a value corresponding to an unnecessarycomponent amount. The unnecessary component amount map 123 thus obtainedis stored in the storage unit 108, for example.

The shift amount d of the second region of interest rb at the time whenthe sum of absolute differences SAD is minimized is not limited to aninteger multiple of the size of the unit pixel 115. Based on a change insum of absolute differences SAD with a change in shift amount d asillustrated in FIG. 12D, the shift amount of the second region ofinterest rb with which the sum of absolute differences SAD is minimizedmay be estimated with an accuracy that is finer than the size of theunit pixel 115. Moreover, as illustrated in FIG. 12B, when the sum ofabsolute differences SAD is hardly changed when the second region ofinterest rb is shifted within the predetermined range, it is consideredthat a large difference corresponding to the parallax is not generatedbetween the plurality of viewpoint images 120 a and 120 b at the spot.Therefore, in such case, the shift amount d of the second region ofinterest rb at the time when the sum of absolute differences SAD isminimized may be treated as 0.

In Step S507, as in Step S306 described above with reference to FIG. 10,the image processing unit 104 smooths a reduced intensity valueDEC(x,y). In other words, the image processing unit 104 applies ageneral smoothing filter or the like to the reduced intensity value mapso that the reduced intensity value is gradually changed at a boundarybetween the unnecessary component generating region GST and the regionin which the unnecessary component is not generated. As a result, asmoothed reduced intensity value DEC′(x,y) is obtained.

In Step S403, the image processing unit 104 performs processing ofreducing the unnecessary component on an image G(x,y) taken byphotography. The image G(x,y) taken by photography is an image generatedfrom the combined signal of the first image signal acquired by the firstpixels G1 and the second image signal acquired by the second pixels G2.When the processing of reducing the unnecessary component is performed,the reduced intensity value obtained in Step S507, that is, the smoothedreduced intensity value DEC′(x,y) is used. The processing of reducingthe unnecessary component is expressed by the following expression (8),for example. In the expression (8), G(x,y) represents an image taken byphotography, and G′(x,y) represents an image on which the processing ofreducing the unnecessary component has been performed.

G′(x,y)=G(x,y)−GHOST(x,y)×DEC′(x,y)/255  (8)

In this manner, the image processing unit 104 reduces the unnecessarycomponent based on the difference between the first viewpoint image 120a and the second viewpoint image 120 b at the time when the correlationvalue is maximized. The processing of reducing the unnecessary componentis selectively performed on the unnecessary component generating regionGST of the image taken by the photography. The processing of reducingthe unnecessary component is performed using the smoothed reducedintensity value DEC′(x,y), and hence, more precisely, this processing isperformed on the unnecessary component generating region GST and thevicinity of the unnecessary component generating region GST.

FIG. 20 is a diagram for illustrating an image in which the unnecessarycomponent is reduced. As illustrated in FIG. 20, in an image 120 c inwhich the unnecessary component is reduced, the unnecessary component,that is, ghost or the like is sufficiently reduced.

As described above, in the second embodiment, the unnecessary componentis reduced based on the difference GHOST(x,y) between the firstviewpoint image 120 a and the second viewpoint image 120 b at the timewhen the correlation value between the first region of interest ra andthe second region of interest rb is maximized. Therefore, according tothe second embodiment, a good image in which the unnecessary componentis reduced can be obtained even when the spot at which the differencecorresponding to the parallax appears conspicuously and the unnecessarycomponent generating region GST overlap with each other.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

For example, in the above-mentioned first and second embodiments, therehas been described, as an example, the case where the image processingapparatus 100 is an image pickup apparatus, but the image processingapparatus 100 is not limited to the image pickup apparatus. For example,the image processing apparatus 100 may be a personal computer (PC), or asmart phone, which is an electronic device having both the function of apersonal digital assistant (PDA) and the function of a mobile phone.Alternatively, the image processing apparatus 100 may be a tabletterminal, a personal digital assistant (PDA), an image viewer, a digitalphoto frame, or an electronic book reader, for example.

This application claims the benefit of Japanese Patent Application No.2016-185868, filed Sep. 23, 2016, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus, comprising: one ormore processors; and a memory storing instructions which, when theinstructions are executed by the processors, cause the image processingapparatus to function as: a detection unit configured to detect anunnecessary component generating region, which is a region of an imagein which an unnecessary component is generated, based on a firstviewpoint image and a second viewpoint image with different viewpoints,the image being obtained by combining the first viewpoint image and thesecond viewpoint image, the detection unit being configured to detectthe unnecessary component generating region based on a plurality ofcorrelation values between a first region of interest in the firstviewpoint image and a plurality of second regions of interest in thesecond viewpoint image; and a reduction unit configured to performprocessing of reducing the unnecessary component.
 2. The imageprocessing apparatus according to claim 1, wherein the detection unit isconfigured to detect the unnecessary component generating region basedon the first viewpoint image and the second viewpoint image that are notcorrected for an image shift, which is generated between the firstviewpoint image and the second viewpoint image.
 3. The image processingapparatus according to claim 1, wherein the plurality of second regionsof interest are set by sequentially shifting one of the plurality ofsecond regions of interest in a predetermined range including a regioncorresponding to the first region of interest.
 4. The image processingapparatus according to claim 1, wherein the correlation value is a sumof absolute differences between a portion located in the first region ofinterest of the first viewpoint image and a portion located in each ofthe plurality of second regions of interest of the second viewpointimage.
 5. The image processing apparatus according to claim 1, whereinthe detection unit is configured to detect the unnecessary componentgenerating region based on comparison between each of the plurality ofcorrelation values and a threshold value.
 6. The image processingapparatus according to claim 5, wherein the threshold value is set basedon a contrast in the first region of interest.
 7. The image processingapparatus according to claim 5, wherein the threshold value is set basedon an average value of pixel values in the first region of interest. 8.The image processing apparatus according to claim 5, wherein thethreshold value is set based on an ISO sensitivity at a time ofphotography.
 9. The image processing apparatus according to claim 1,wherein the reduction unit is configured to reduce the unnecessarycomponent so that a degree of reduction of the unnecessary component isgradually changed at a boundary between the unnecessary componentgenerating region and an unnecessary component non-generating region,which is a region in which the unnecessary component is not generated.10. The image processing apparatus according to claim 1, wherein thereduction unit is configured to reduce the unnecessary component basedon a difference between the first viewpoint image and the secondviewpoint image at a time when one of the plurality of correlationvalues is maximized.
 11. The image processing apparatus according toclaim 1, wherein the reduction unit is configured to selectively performthe processing of reducing the unnecessary component on the unnecessarycomponent generating region.
 12. An image processing method, comprising:detecting an unnecessary component generating region, which is a regionof an image in which an unnecessary component is generated, based on afirst viewpoint image and a second viewpoint image with differentviewpoints, the image being obtained by combining the first viewpointimage and the second viewpoint image, the unnecessary componentgenerating region being detected based on a plurality of correlationvalues between a first region of interest in the first viewpoint imageand a plurality of second regions of interest in the second viewpointimage, which are located in a vicinity of a region corresponding to thefirst region of interest; and performing processing of reducing theunnecessary component.
 13. A non-transitory computer-readable storagemedium having stored thereon a program for causing a computer toexecute: detecting an unnecessary component generating region, which isa region of an image in which an unnecessary component is generated,based on a first viewpoint image and a second viewpoint image withdifferent viewpoints, the image being obtained by combining the firstviewpoint image and the second viewpoint image, the unnecessarycomponent generating region being detected based on a plurality ofcorrelation values between a first region of interest in the firstviewpoint image and a plurality of second regions of interest in thesecond viewpoint image, which are located in a vicinity of a regioncorresponding to the first region of interest; and performing processingof reducing the unnecessary component.